Regression testing of software is used to test changes to computer programs to make sure that older programming still works when new changes have been made. This is part of the normal program development process, required because changing or adding new code to a program can easily introduce errors into code that it is not intended to change. Regression testing comprises the development of code test scenarios and exercises to test new units of code after they have been written. Before a new version of a software product is released, the old test cases are run against the new version to make sure that all the old capabilities still work.
In the main these techniques involve manually writing tests which are run and then followed up with a manual observation of the results to detect success or failure. What is tested is dependant upon the resources available and the time within which this validation has to be performed. In manual testing one essentially has an operator manually running activities via an interface device to the system under test. This can be automated by imposing an additional system which can run a sequence of activities, and which can be a Web initiator, a Personal Computer, or any programmatic device.
In addition, the test environment will want to ensure that all operations are at least occasionally executed (to ensure full coverage). Prior art techniques, such as that disclosed in U.S. Pat. No. 6,694,509, attempt to overcome this problem, sometimes called ‘missing function’, where some part of the software is supposed to be tested but actually is omitted. The prior art techniques make alterations to the test suite while the system is offline or stopped. The system can then carry out further tests in a different area or manner to enforce a test structure according to the items being tested. These techniques are essentially coded versions of manual tests and do not exhibit any degree of automation apart from non-manual execution.
The present invention is based on state-based testing methods, sometimes called model-based testing, such as that described in U.S. Pat. No. 6,408,262. State-based programming is a common methodology for exploring the circumstances and testing of computer systems. In this technique, a set of states, each of which corresponds to a particular circumstance, and operations on these states, which move the system from one state to another or back to the same state, are defined. Note that what constitutes a state, or how the state of a system is determined, is an implementation-dependent variable. For example, software could announce its state; actions could be taken to ask for the current state; or actions could be taken to determine the state by reference to various externals to the chunk of software being considered.
An example of a state table is shown in Table 1:
Starting state:State AState BState CState DAction 1ADCBAction 2DDDAAction 3CBCD
There are four possible states listed: A, B, C & D, each corresponding to a particular well-defined state of activity in the system under test. For the purposes of this explanation, it is irrelevant what these states actually correspond to, only that they exist and cover all the possible situations. The table also lists actions which correspond to all the possible actions that the system supports—three in this case 1, 2 & 3. Again, what these actions actually are is not relevant, but they cover all the possible actions that the system supports. In some cases, certain actions will not be valid for certain states, which means that the system would not be expected to take that action from that state.
Table 1 shows that if the software is in state A, action 1 will not actually cause a state change—the system remains in state A. However, if action 2 is executed instead of action 1, the state of the software system will move into State D. As the table shows, some states can never be entered from certain other states—for example, state C cannot be entered from state B, as none of the actions support this operation. Conversely, some actions will never cause a certain state to be entered, e.g. action 2 will never cause state C to be attained. If after executing action 2 state C is attained, an error has occurred. Similarly, if action 3 is executed while in state B and the software system ends up in state D, an error has occurred. Known techniques for processing based on state tables can easily detect failures.
A finite state diagram showing the various states and the possible transitions there-between, according to state table 1, is shown in FIG. 1A.
A state table describing a model of a system may include states such as ‘stopped’, from which the execution of any action should not cause a state change, as well as ‘indeterminate’, which indicates that an error has occurred and the state of the system cannot be ascertained.
In Table 1, there are four possible states and three possible actions. Thus, starting from a known point (say state 1) the system supports 4*3 things to be evaluated. However, one also potentially wants to start from one of the other three states as well, so there are potentially 4*4*3=48 items to test. This assumes that a single pass through the table is deterministic and always performs for the nth pass as it does on the first pass. This is, in general, an unwise assumption and so multiple passes must be executed over a long time (to test out, for example, timing or resource constraints etc.). This repetition is especially important for Regression Testing.
Some testing methods predefine one or more sequences of actions (tests) to be carried out during testing. Actions are executed in the pre-defined order, the results are examined and the process stops if something unexpected occurs. Consider the following series of discrete test runs:
A B K; A B K; A C; A B K; A D H; A B L; A C
where each letter represents a different test and a semi-colon represents a re-start between runs of the testing process. Such test sequences are generated before testing begins, either randomly or on the basis of some type of failure. For example, if test K failed during the series described above, then the functionality generating test sequences (again, one fixed sequence for each run) could bias attempts to do more K function by coming up with something like:A C; A B K; A B K; B K J; A C; B E; A B K; A B K; A B L; B K JAnalysis of the results of each of these testing sequences (occurring in separate test runs) is done to more precisely locate the bug which caused K to fail.
After each test run, the results are examined and if something has happened which is unexpected, processing stops. Some off-line analysis is done to determine the cause of the failure, and a new series of operations prepared for the next run of the testing.
Other testing methods use random selection of actions to be applied from the set of available actions with the selection of tests for a sequence of test runs being carried out before the sequence is started. In this case, each run will not cause exactly the same sequence of test operations to be executed. This is better as it is, to a limited extent, a non-repeatable sequence (over time). However, in these methods testing is still halted when an error occurs.
Note that for a predefined test run the starting state is defined. This means that operations must be carried out on the system to achieve the required starting state before testing begins.
It has been observed that software bugs tend to cluster around each other. Thus, if one bug is found there is a chance that there are others nearby which should also be detected. Prior art techniques rely on restructuring a sequence of tests offline and rely heavily on human interaction to enable investigation around a discovered bug.
The present invention aims to overcome the above problems and to improve the detection of errors.